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FOREWORD 


No  one  questions  the  need  for  quality  training  programs.  However,  it 
is  very  appropriate  that  we  attempt  to  get  the  most  value  out  of  our 
training  ’'dollar."  As  a matter  of  fact,  this  phase  of  our  training 
effort  is  long  overdue. 

Here  is  a report  of  an  attempt  to  develop  a method  of  training  eval- 
uation that  is  closely  identified  with  operational  requirements  and 
active  supervisory  responsibility.  It  was  designed  by  our  Personnel 
Research  Staff.  It  can  be  implemented  at  any  organizational  level. 

This  report  contains  information  that  raises  pointed  questions.  Care- 
ful study  of  questions  like  these  can  show  us  how  to  overcome  short- 
comings in  our  training  effort,  and  how  our  total  training  program  can 
make  its  most  effective  contribution. 

In  addition  to  Problem  Analysis  and  Decision-Making  training,  we  are 
already  applying  this  method  to  the  SED  and  SIMM  programs.  All  of 
you  who  have  a share  in  training  efforts  will,  I am  sure,  see  other 
ways  in  which  this  method  can  be  put  to  work  to  point  up  more  ques- 
tions and  develop  more  information  that  can  be  of  value  to  you. 
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SUMMARY  AND  HIGHLIGHTS 


The  purpose  of  this  research  was  to  develop  a method  which  could  serve 
as  a prototype  for  evaluating  managerial  training.  In  addition  to  exemplify- 
ing the  general  rules  underlying  all  training  evaluation,  the  procedure  was 
designed  to  test  the  effectiveness  of  the  training  effort  against  specific 
requirements  of  a sound  training  program.  These  requirements  include  aware- 
ness by  participants  and  their  supervisors  of  the  specific  objectives  of  the 
proposed  training,  encouraging  continuing  attention  to  employee  development 
needs,  the  integration  of  training  with  operations,  and  adequate  follow-up 
by  supervisors  as  well  as  by  training  staffs. 

Two  sets  of  two  questionnaires  each  were  constructed.  One  set  was  for 
participants;  the  other  for  supervisors.  Each  set  included  a questionnaire 
to  be  administered  before  training,  and  another  for  use  immediately  after 
training.  The  questionnaires  were  administered  in  connection  with  the 
Kepner-Tregoe  course  in  Management  Problem  Analysis  and  Decision  Making, 

Of  a total  of  54  participants  in  five  separate  sessions  of  the  course,  53 
returned  completed  questionnaires.  Returns  from  supervisors  were  approxi- 
mately 75%,  Responses  of  both  groups  were  content-analyzed  and  classified 
according  to  level  or  quality  of  content, 

RESPONSES  TO  PRE -TRAINING  QUESTIONS 

Most  participants  and  supervisors  had  more  or  less  clearly  expressed 
ideas  as  to  the  objectives  of  the  course,  but  only  19%,  of  the  super- 
visors and  26%  of  the  participants  identified  these  objectives  with 
the  work  of  the  participant  or  his  value  as  an  employee. 

13%  of  the  participants  and  26%,  of  the  supervisors  gave  responses 
that  reflected  no  awareness  of  career  plans  or  career  development, 

26%  of  participants  and  43%  of  supervisors  were  able  fco  mention 
specific  problems  which  they  hoped  would  be  handled  more  effectively 
by  the  participant  after  training,  34%  and  24%,  respectively,  were 
unable  to  point  to  any  specific  problem  or  problem  area  to  which  they 
thought  the  training  might  be  applicable. 

In  approximately  50%  of  the  cases  no  discussion  took  place  between 
participant  and  supervisor  prior  to  training. 

About  30%  of  both  groups  cited  specific  incidents  of  job  performance 
that  pointed  up  the  need  for  this  kind  of  training.  34%  of  partici- 
pants and  47%  of  supervisors  failed  to  present  any  specific  evidence 
of  training  needs. 

73%  of  supervisors  and  62%  of  participants  considered  the  advance 
information  concerning  the  course  adequate.  27%  and  38%,  respectively, 
felt  that  it  was  insufficient  or  non-existent. 
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Approximately  25%  of  the  responses  explicitly  mentioned  the  parti- 
cipant as  taking  part  in  the  decision  that  he  attend  the  course. 

11%  of  the  participants  were  unable  to  say  who  had  taken  part  in 
the  decision. 

RESPONSES  TO  POST-TRAINING  QUESTIONS 

21%  of  the  supervisors  and  58%  of  the  participants  reported  changes 
in  their  ideas  concerning  the  objectives  of  the  course. 

The  percentage  of  participants  who  were  able  to  suggest  specific 
applications  in  their  regular  jobs  increased  from  26%  before  train- 
ing to  46%  after  training.  The  percentage  who  were  unable  to  visu- 
alize any  useful  applications  of  the  course  dropped  from  34%  to  14%. 

Of  42  participants  for  whom  complete  data  were  available,  50%  were 
able  both  to  specify  applications  of  the  training  and  to  cite  situa- 
tions which  they  would  have  handled  differently  if  they  had  already 
taken  the  course.  7%  were  unable  to  indicate  either  applications  or 
situations  that  they  would  have  handled  differently. 

Most  participants  and  supervisors  indicated  some  plans  for  mutual 
discussion  and  follow-up  regarding  the  results  of  the  training, 

QUESTIONS 

Is  there  sufficient  communication  between  training  staff  and  line 
supervisors  concerning  the  content  and  objectives  of  available 
courses? 

Is  the  supervisor  usually  provided  with  enough  information  to  enable 
him  to  make  the  best  choices  among  courses  and  among  potential 
trainees? 

If  a supervisor  is  unable  to  document  specific  training  needs,  or 
to  specify  ways  in  which  he  expects  employees  to  apply  the  proposed 
training,  on  what  basis  does  he  select  trainees? 

Is  too  heavy  reliance  placed  upon  the  "quota81  method  of  assignment? 
Can  methods  of  selecting  participants  be  improved? 

Is  the  mere  hope  that  an  employee  will  be  "helped"  by  a course,  about 
which  the  supervisor  knows  little  more  than  its  title,  enough  justi- 
fication for  the  cost  of  the  training? 

Do  supervisors  sufficiently  discuss  the  proposed  training  with  their 
employees? 
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Is  the  awareness  of  the  various  responsibilities  for  management 
development,  held  and/or  shared  in  each  organization,  as  clear  as 
it  might  be? 

Would  the  use  of  this  kind  of  evaluation  procedure  on  a systematic 
and  continuing  basis  result  in  a "tightening  up"  of  the  training 
effort  all  along  the  line?  Would  such  an  outcome  be  desirable? 
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A PATH  TO  MANAGEMENT  DEVELOPMENT  AND  TO 
THE  MEASUREMENT  OF  ITS  GROWTH1 


I.  Introduction 


There  is  no  denying  the  fact  that  training,  particularly  management 
training,  is  most  difficult  to  appraise.  Tangible  effects  directly  attri- 
butable to  such  training  are  quite  elusive,  and  they  may  occur  at  a time 
relatively  remote  from  the  course  of  instruction. 

But  lack  of  reasonably  objective  evidence  makes  those  of  us  who 
sponsor  and  support  management  training  vulnerable  to  our  critics,  who 
like  to  see  evidence  in  physical  or  fiscal  terms.  Also,  among  those 
responsible  for  training,  the  deficiency  of  solid  data  makes  some  unduly 
susceptible  to  enthusiasm,  while  others  hesitate  to  launch  or  subscribe 
to  potentially  good  programs  for  fear  that  they  might  be  unable  to  justify 
their  judgment  adequately. 

The  measure  of  effective  training  is  change-change  in  performance 
that  contributes  to  more  effective  mission  accomplishment  for  the  organi- 
zation sponsoring  the  training.  First  and  foremost,  of  course,  we  record 
changes  in  the  man.  But  this  is  not  enough,  if  we  are  to  appraise  the 
success  of  the  training  effort  in  terms  of  the  total  outcome.  In  our 
intense  concentration  upon,  "employee  development"  we  often  lose  sight  of 
the  fact  that  employee  development  is  an  abstraction  unless  it  is  care- 
fully and  systematically  integrated  with  the  overall  requirements  of 
"management  development."  Hence,  the  effects  that  we  look  for  should  be 
reflected  not  only  in  changes  in  the  man,  but  also  in  his  supervisor,  in 
higher  management,  and  in  the  total  work  situation.  The  training  of 
managers  is  but  one  phase  of  management  development. 

Are  the  supervisor  and  the  management  ready  for  the  "changed  man?" 
This  question  is  not  often  asked  prior  to  assigning  an  individual  to 
training.  It  may  seem  like  a silly  question,  because  it  is  "self-evident" 
that  an  employee  would  not  be  sent  to  a course  unless  it  was  expected  that 
he  would  learn  something  of  benefit  to  the  organization.  However,  some 
interesting  research  has  disclosed  that  a man  may  not  be  able  to  use  the 
knowledge  gained  because  it  involves  changes  in  the  man  or  in  the  situa- 
tion that  his  associates  are  not  used  to,  and  deliberately  or  unwittingly 
are  unable  to  accept.  In  some  cases  it  has  even  been  shown  that  the  man 
is  more  successful  in  applying  what  he  has  learned  if,  after  training,  he 
goes  to  a different  shop  or  office  than  the  one  he  started  out  from.  In 
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^This  report  describes  the  first  phase  of  a research  project  under- 
taken at  the  request  of  the  Employee  Development  Division  of  the  Office 
of  Personnel. 


2 


the  new  place,  the  established  conceptions  as  to  how  Joe  is  suppose  to 
act  and  how  he  is  to  function  are  not  present  to  hobble  him. 

It  holds  from  this  point  of  view  that  benefits  from  training  are  a 
function  not  only  of  the  nature  and  degree  of  participation  and  experi- 
ence of  the  employee-trainee,  but  also  of  the  role  played  by  supervisors 
and  managers,  as  well  as  the  interaction  among  all  parties  involved  in 
planning  for  training  and  for  its  application. 

This  was  the  orientation  with  which  we  approached  the  problem  of 
developing  a.  prototype  system  for  more  effectively  managing  and  evaluating 
managerial  training.  And  from  this  orientation  derived  the  statements  of 
specific  objectives  of  this  research  effort  (an  action  research  program  in 
part),  and  of  the  outcomes  of  training  expected  to  serve  as  a basis  for 
judging  the  degree  of  its  success. 
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II.  What  We  Looked  For 


General  Requirements 

Five  general  rules  guided  the  present,  effort.  These  rules  are  best 
exemplified  by  questions  which  serve  to  define  the  requirements  that  the 
evaluation  must  satisfy. 

1.  What  specific  things  is  the  trainee  expected  to  do,  or  to  do 
differently  as  a result  of  this  training  experience?  The  effectiveness 
of  any  training  program  can  only  be  assessed  in  terms  of  the  specific 
objectives  of  that  program.  Furthermore,  statements  of  objectives  must 
always  point  toward  observable  behaviors  or  behavior  changes,  if  evalua- 
tion is  to  be  possible. 

2.  How  did  the  individual  act  before  he  was  trained?  How  did  he 
act  after  training?  Since  the  effects  of  training  appear  as  changes  in 
behavior,  it  is  necessary  to  have  a "bench  mark"  to  know  where  we  started 
from. 


3.  Are  the  long-term  effects  the  same  as  the  more  immediate  changes 
in  behavior?  Often  the  two  pictures  will  lead  to  quite  different  con- 
clusions . 


4.  What,  precisely,  do  we  wish  to  evaluate-- the  content  of  the  pro- 
gram, the  method  of  teaching  it,  the  ability  of  the  instructors,  or  what? 
For  present  purposes  it  seemed  most  desirable  to  appraise  the  combined 
impact  of  all  such  factors,  rather  than  try  to  identify  the  specific 
effects  of  each  separately. 


5 What  use  is  to  be  made  of  the  evaluation  process?  A procedure 
like  the  one  to  be  described  has  many  possible  applications.  It  can 
provide  an  objective  appraisal  of  the  effectiveness  of  a particular  course. 
Used  systematically  and  on  a continuing  basis,  it  can  be  a means  of  tracing 
changes  in  the  effectiveness  of  the  training  program  over  a period  of  time. 
By  pinpointing  specific  weaknesses  either  in  a training  program  or  in  its 
administration,  it  can  lead  eventually  to  better  results  from  the  training 
effort.  The  understanding  that  a follow-up  is  expected  to  be  a regular 
part  of  the  training  process  should  stimulate  participants  and  their  super- 
visors to  think  more  carefully  about  any  program  that  is  under  considera- 
tion, and  more  consistently  about  training  needs  and  employee  development 
as  a continuing  responsibility.  More  discriminating  selections  both  of 
courses  and  participants  would  represent  a substantial  gain  in  itself. 


Requirements 


What  are  the  desirable  features  of  a sound  training  program,  the 
things  to  look  for  in  planning  an  evaluation?  The  following  list  of 
specifics  is  not  necessarily  exhaustive;  however,  it  covers  the  main 
points  which  determined  our  approach  to  the  task: 
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1.  Both  participants  and  supervisors  should  display  some  advance 
knowledge  of  the  specific  objectives  of  the  training.  This  should  be 
combined  with  evidence  of 

2.  Knowledge  of  the  employee’s  individual  training  needs,  and 

3.  Perception  of  the  role  of  this  particular  training  in  relation 
to  his  career  development.  At  the  same  time, 

4.  Training  should  be  integrated  with  operations--supervisors  and 
participants  alike  should  be  able  to  indicate  definite  ways  in  which  the 
employee  is  expected  to  use  the  training  in  his  work.  This  implies 

5.  Communication  between  supervisor  and  employee,  indicating  that 
both  are  involved  in  planning  the  training  effort.  It  also  calls  for 

6.  Adequate  feedback  to  the  supervisor,  as  well  as  to  agency  and 
departmental  training  staffs. 
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III.  WHAT  WE  DID 


An  essential  requirement  of  the  prototype  was  that  it  must  be  appli- 
cable to  many  different  training  programs.  This  requirement  obviously 
eliminates  the  possibility  of  laying  down  any  single  set  of  performance 
criteria  against  which  all  programs  can  be  appraised.  Actually,  this  is 
not  a serious  handicap  in  view  of  what  we  were  looking  for.  The  charac- 
teristics of  a good  program,  as  just  outlined  are  common  to  all  training, 
as  are  the  requirements  of  feedback  to  supervisors  and  management.  At 
the  same  time  we  may  assume  that  both  employee  and  supervisor  can  recog- 
nize effective  and  ineffective  job  performance,  that  they  can  translate 
the  latter  into  expressions  of  training  need  and,  finally,  that  they  can 
recognize  changes  in  performance  when  or  if  they  occur. 

To  try  out  the  technique,  a five-day  course  in  "Management  Problem 
Analysis  and  Decision-Making"  was  selected, ^ This  course  is  offered  on  a 
contract  basis  to  government  agencies  and  industrial  firms  by  Kepner,  Tregoe 
and  Associates,  Inc.,  of  Princeton,  New  Jersey,  During  1962  a number  of 
sessions  were  held  for  USDA  employees  in  GS  grades  12  to  15.  Conducted  as 
a workshop  and  seminar,  the  course  teaches  specific  concepts  and  methods 
which  appear  to  be  directly  applicable  in  the  trainee’s  daily  work. 

Two  sets  of  two  questionnaires  each  were  constructed  (Appendix  A) . One 
set  was  for  trainees;  the  other  for  their  supervisors.  Each  set  included 
one  questionnaire  to  be  administered  prior  to  training;  and  another,  imme- 
diately after  training.  A third  pair  of  questionnaires  will  be  used  six 
months  or  more  following  completion  of  the  training.  This  final  follow-up 
will  be  described  in  a later  report. 

"Before"  and  "After"  questionnaires  were  given  to  all  participants  in 
five  separate  sessions  of  the  Kepner-Tregoe  course  held  between  October  15 
and  December  14,  1962,  The  total  group  consisted  of  54  individuals,  of 
whom  53  returned  the  information  requested.  Exhibits  1 and  2 show  the  dis- 
tributions of  these  participants  by  USDA  agencies  and  by  grade  levels, 
respectively. 

Pre-training  questionnaires  were  mailed  to  the  supervisors  of  all  par- 
ticipants. Post-training  questionnaires  were  sent  to  supervisors  of  the  44 
participants  in  the  last  four  sessions  of  the  course--the  first  session  was 
not  represented  because  this  questionnaire  was  not  developed  in  time.  A 
return  of  approximately  7 57,  was  obtained  in  both  cases. 

The  responses  to  each  question  were  sorted  into  categories  according 
to  the  level  of  quality  of  the  content.  No  more  than  three  categories  were 
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See  Kepner,  Charles  H,  and  Tregoe,  Benjamin  B.,  Developing  decision 
makers.  Harvard  Bus iness  Review,  1960,  _38,  115-124,  for  a brief  account  of 
the  course 
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used  in  any  case.  Each  category  was  scored  2,  1,  or  0;  the  higher  number 
always  indicating  the  "better"  or  preferred  answer,  as  will  be  illustrated 
shortly.  When  the  question  was  ot  the  "yes-no"  type,  only  two  were  needed. 


Exhibit  1 

Distribution  of  Participants  by  USDA  Agencies 


Agency 

Number  of 
Participants 

Forest  Service 

36 

Agricultural  Marketing  Service 
Agricultural  Stabilization  and 

5 

Conservation  Service 

5 

Agricultural  Research  Service 

4 

Farmer  Cooperative  Service 

1 

National  Agricultural  Library 

1 

Soil  Conservation,  Service 

1 

Total 

53 

The  categories  used  in  classifying  the  answers  to  each  question  were 
determined  primarily  by  content  analysis.  It  was  relatively  easy  in  this 
way  to  group  a.  given  set  of  answers  under  two  or  three  broad  themes  which 
were  relevant  to  the  objectives  of  the  study.  The  classification  schemes 
thus  developed  were  tested  for  objectivity  by  checking  the  agreement 
between  sorters  who  worked  independently.  When  members  of  the  Personnel 
Research  Staff  sorted  random  samples  of  answers  in  this  way,  agreement 
ranged  between  80%  and  1007o  from  one  question  to  another. 


Exhibit  2 

Distribution  of  Participants  by  Grade  Level 


GS  Level 


Number  of 
Participants 


15 

14 

13 

12 


1 

27 

16 

9 


Total 


53 
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IV.  What  We  Found 


Pre-Training 

The  pre-training  questionnaires  for  both  participants  and  super- 
visors contain  seven  main  items,  plus  an  eighth  which  invites  "other 
comments."  Since  the  two  sets  of  questions  are  generally  parallel, 
content  analyses  yield  the  same  categories  for  both  sets.  These 
categories  will  now  be  presented  for  each  question,  in  turn,  together 
with  a. summary  of  their  frequency  of  occurrence  among  participants 
and  supervisors. 
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Question  Pre-1:  WHAT  DO  YOU  THINK  ARE  THE  SPECIFIC  OBJECTIVES  OF  THE 
COURSE? 


Level  2 - Responses  containing  explicit  reference  to  the  participant  in 
terms  of  applying  the  training  in  his  present  or  future  work,  or  improving 
his  value  to  the  organization. 

"To  increase  my  ability  and  efficiency  as  an  administrator  by  training 
in  methods  of  problem  solving  and  decision-making." 

"To  give  this  participant  knowledge  which,  if  applied,  will  enable  him 
to  make  a greater  percentage  of  right  decisions,  thus  improving  his 
performance  and  increasing  his  value  to  his  employers." 

Level  ^ - General  statement  without  explicit  reference  to  participant's 
present  work  of  future  value  to  the  organization. 

"Help  managers  do  a better  job  of  solving  problems  and  making  decisions." 

"To  improve  the  skill  of  analyzing  problems  and  making  decisions." 

Level  0 - General  statement  without  explicit  reference  to  either  problem 
analysis  or  decision-making. 

"The  course  will  present  a systematic  method  of  reasoning." 

"To  assist  participants  in  understanding  the  elements  they  use  in  meet- 
ing everyday  work  situations." 
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Exhibit  3 


Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-1 
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Exhibit  3 shows  the  percentages  of  participants  and  of  supervisors 
whose  answers  to  Question  Pre-1  fell  in  each  of  the  three  categories. 

The  two  groups  were  much  alike  in  their  understanding  of  the  objec- 
tives of  this  particular  course.  The  number  who  had  no  idea  of  the 
objectives  was  negligible.  Most  had  a more  or  less  clearly  expressed 
idea  in  their  answers  (Level  1),  but  only  19%  of  the  supervisors  and 
26%  of  the  participants  explicitly  identified  these  objectives  with  the 
participant's  actual  work  or  his  value  as  an  employee.  Others,  no 
doubt,  took  this  identification  for  granted;  nevertheless  it  would  have 
been  suggestive  of  more  careful  thought  and  planning  if  they  had  ex- 
pressed the  idea  in  so  many  words. 
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Question  Pre-2:  HOW  DOES  THIS  SEEM  TO  FIT  INTO  YOUR  OWN  CAREER  PLANS? 
(Participants) 

HOW  DOES  THIS  FIT  INIO  YOUR  AGENCY  TRAINING  AND  DEVELOP- 
MENT PROGRAM  FOR  THIS  PARTICIPANT?  (Supervisors) 


Leve 1 2 - Response  contains  some  allusion  to  long-range  plans,  e.g., 
continuing  in  present  career,  changing  career  field,  or  advancing 
within  career,  (Participants) 

Response  contains  explicit  allusion  to  potential  of  participant, 
either  in  his  present  position  or  in  future  assignments.  (Supervisors) 

"In  my  position  the  training  should  help  toward  better  redeeming 
my  responsibilities  as  a manager,  thereby  enhancing  opportunity 
for  continuing  up  the  career  ladder." 

"This  participant  apparently  has  a high  potential  as  a research 
administrator.  His  ability  will  be  needed  in  expanding  USDA 
program.  This  training  was  assigned  to  increase  his  ability  to 
handle  his  present  and  future  assignments." 

Leve 1 1 - Response  refers  only  to  immediate  demands  of  present  job. 
(Participants) 

Response  indicates  that  course  is  considered  part  of  agency's 
regular  training  program,  but  contains  no  specific  reference  to  parti- 
cipant's potential  or  his  career  plans.  (Supervisors) 

"I  am  a branch  chief  with  nation-wide  programs  calling  for  frequent 
decisions.  I expect  this  course  to  be  useful  to  me  in  running 
these  programs." 

"Our  Division  has  a formalized  training  program  which  calls  for 
this  employee  to  take  courses  which  will  provide  across  the  board 
management  training.  This  particular  course  will  fill  a part  of 
the  requirements  of  this  program." 

Level  ()  - No  specific  tie-in  either  with  present  work  or  with  future 
plans.  (Participants) 

General  statement  as  to  effectiveness  of  and  need  for  such 
training,  but  no  specific  description  of  its  role  in  participant's 
career  development.  (Supervisors) 

"Fits  in  well." 

"A  supervisor's  deputy  should  analyze  problems  and  make  decisions. 
Ability  to  do  so  is  essential  to  the  job." 
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Exhibit  4 

Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-2 


SUPERVISORS 


The  percentage  distributions  of  answers  to  Question  Pre-2  are 
shown  in  Exhibit  4. 

It  is  interesting  that  45%  of  the  participants,  but  only  19%  of 
the  supervisors  responded  at  Level  2.  This  might  suggest  that,  as  a 
group,  employees  have  a clearer  picture  of  their  career  plans  and  of 
the  role  of  this  training  in  relation  to  these  career  plans  than  do 
their  supervisors . Supervisors  as  a group  seem  to  have  had  a tendency 
to  think  of  the  course  more  in  terms  of  formally  prescribed  training 
programs  than  in  terms  of  individual  career  plans  (55%  of  answers 
scored  at  Level  1).  The  apparent  difference,  however,  may  be  an  arti- 
fact, since  the  supervisors'  question  is  not  explicitly  career-oriented. 
In  any  event,  it  may  be  disconcerting  to  note  that  13%  of  the  partici- 
pants and  26%  of  the  supervisors  gave  answers  that  reflected  no  aware- 
ness of  career  plans  or  career  development.  These  differences  between 
supervisors  and  participants  are  highly  significant  statistically. 

That  is,  they  are  too  large  to  be  attributed  to  chance . ^ 


■^A  two-tailed  sign  test  was  significant  at  the  .04%  level  of 
confidence . 
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Question  Pre-3.  ON  THE  BASIS  OF  INFORMATION  AVAILABLE  TO  YOU  NOW,  IN 

WHAT  PARTICULAR  WAYS  DO  YOU  EXPECT /EXPECT  THIS  EMPLOYEE 
TO  APPLY  THIS  TRAINING  IN  YOUR/HIS  FUTURE  WORK?  GIVE 
EXAMPLES . 


Level.  2 - Response  mentions  specific  task(s)  or  problem(s)  , 

"I  expect  the  employee  to  use  the  material  to  determine  what  is  the 
cheapest  and  most  effective  technique  to  use  in  controlling  a speci- 
fic fire." 

"Help  decide  between  (a)  establishment  of  a study  to  limit  of  present 
capabilities  or  work  to  get  support  of  more  adequate  study;  (b)  set 
up  program  of  research  in  our  own  organization  or  contract  with 
another  agency  " 

Leve  1 1_  - Reference  to  general  task  or  problem  area,  but  without  specific 
examples  „ 

"In  developing  regulations  and  instructions  for  new  programs  for 
which  I have  been  given  responsibility;  in  analyzing  questions  and 
problems  presented  by  field  offices  with  respect  to  these  new  pro- 
grams and  also  programs  which  are  my  primary  responsibility." 

"This  employee  is  in.  effect  our  Public  Relations  expert.  We  look 
to  him  for  guidance  as  to  how  our  agency  can  best  carry  out  its 
program  with  public  acceptance  and  support  This  involves  careful 
analysis  of  many  factors,  often  not  immediately  apparent  as  impor- 
tant-how this  group  or  that  group  will  react  and  why;  or  can 
management  attain  the  same  objectives  but  in  a way  that  is  more 
acceptable  to  the  affected  public?" 

Level  0 - General  statements  without  reference  to  any  actual  task  or 
problem  area. 

"Making  decisions  is  part  of  the  daily  job  Training  in  good 
decision-making  will  be  used  everyday  " 

"This  employee  will  be  faced  with  problems  national  in  scope  and 
extending  across  all  commodity  lines  on  a day  to  day  basis  If 
it  doesn't  help  him  it  is  no  good  or  mislabeled." 
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Exhibit  5 


Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-3 
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Although  Exhibit  5 shows  some  apparent  differences  between  super- 
visors and  participants  in  their  responses  to  Ouestion  Pre-3,  these 
differences  are  not  statistically  significant.  The  most  that  we  can 
say  here,  therefore,  is  that  while  fairly  sizable  numbers  of  both  par- 
ticipants (267.)  and  supervisors  (437.)  were  able  to  mention  specific 
problems  which  they  hoped  would  be  handled  more  effectively  after  com- 
pletion of  the  course  (Level  2),  considerable  numbers--347.  and  247.-- 
were  unable  to  point  to  any  specific  problem  or  any  problem  area  to 
which  they  thought  the  training  might  be  applicable.  Apparently,  they 
hoped  that  the  participant  might  benefit  from  the  course  in  some  way. 
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Question  Pre-4:  HAVE  YOU  DISCUSSED  POINTS  1,  2,  AND  3 WITH  YOUR 

SUPERVISOR/THIS  EMPLOYEE?  IF  YES,  DO  YOU  FEEL  THAT 
HE  SHARES  THE  SAME  EXPECTATIONS  AS  YOU  DO?  IF  NOT, 
HOW  DO  YOUR  EXPECTATIONS  DIFFER  AS  YOU  SEE  IT? 


Leve 1 2 - "Yes"  to  both  questions. 


Level  _1  - "No"  to  first  question;  "Yes"  to  second. 

Leve 1 0 - "No"  to  first  question,  no  answer  or  "don't  know"  to  second. 


Exhibit  6 


Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-4 
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As  might  be  expected,  Exhibit  6 shows  close  agreement  between  par- 
ticipants and  supervisors.  Practically  no  one  on  either  side  of  the 
fence  reported  feeling  that  his  expectations  were  not  shared.  Perhaps, 
the  most  noteworthy  finding  brought  out  by  this  question  is  that  in 
approximately  half  the  cases,  before  the  training,  no  discussion  took 
place  between  supervisor  and  participant  (only  47%  of  participants  and 
55%  of  supervisors  gave  Level  2 responses). 
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Question  Pre-5:  LOOKING  BACK  OVER  YOUR/HIS  WORK  DURING  THE  PAST  SIX 
MONTHS  OR  SO,  HAS  ANYTHING  HAPPENED  TO  BRING  HOME  TO 
YOU  THE  NEED  FOR  THIS  KIND  OF  TRAINING?  GIVE  ACTUAL 
EXAMPLES . 


Leve 1 2 - Response  contains  example(s)  with  some  specification  of  problem(s) . 

"Difficulty  in  hiring  key  man  in  one  discipline  forced  us  into  con- 
tract with  a university.  Had  I been  willing  and  able  to  read  hand- 
writing on  wall  sooner,  we  could  have  made  a better  'marriage*  and 
gotten  work  underway  sooner." 

"He  sent  a subordinate  on  a mission,  the  results  of  which  were  not 
thought  through,  and  the  mission  had  to  be  called  off.  This  could 
have  been  avoided  by  thorough  consideration  of  the  consequences." 

Leve  1 JL  - Task  or  problem  area  indicated,  but  specific  example(s)  not  given. 

"For  almost  a year  I have  been  evaluating  the  magnitude,  cost  and 
direction  being  taken  in  service -wide  programs.  To  prescribe  policy 
and  procedure  is  the  next  step,  then  evaluate  progress." 

"The  problems  of  federal  land  use  and  administration  with  which  this 
office  deals,  are  becoming  increasingly  complicated.  Values  are  con- 
tinually enhanced,  competition  for  lands  more  intense  This  brings 
to  the  forefront  the  need  for  increasing  technical  competency  and 
accurate  thinking  on  specific  problems," 

Level  0 - General  answer— no  specific  task  or  problem  area  indicated, 

"Nothing  specific.  In  general,  the  need  to  keep  ahead  of  a rapidly 
increasing  activity  with  many  growing  problems," 

"Haven't  been  associated  long  enough  to  know  his  need  for  this  kind 
of  training- -other  than  I know  it  will  prove  helpful." 
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Exhibit  7 


Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-5 
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As  with  Questions  1,  3,  and  4,  supervisors  and  participants  did 
not  differ  significantly  in  the  distributions  of  their  replies  to 
Question  Pre-5.  Exhibit  7 shows  that  approximately  30%  of  both  groups 
cited  specific  incidents  of  job  performance  that  pointed  up  a need  for 
training  in  problem  analysis  and  decision-making.  Another  257«  to  34% 
were  aware  of  general  task  areas  in  which  this  need  appeared.  Once 
again,  however,  a large  proportion — 34%  of  the  participants  and  47% 
of  the  supervisors  — failed  to  present  any  specific  evidence  of  train- 
ing needs. 


17 


COMPARISON  OF  QUESTIONS  PRE-3  AND  PRE-5 


Since  Questions  Pre-3  and  Pre-5  are  somewhat  related,  it  is  informa- 
tive to  compare  the.  distributions  of  responses  0 The  participants  were 
quite  consistent,  as  Exhibit  8 shows.  The  supervisors,  on  the  other  hand, 
showed  a higher  proportion  of  Level  2 answers  on  Question  Pre-3,  and  a 
higher  proportion  of  zero  answers  on  Question  Pre-5 „ These  differences, 
were  significant  statistically This  would  seem  to  indicate  that  super- 
visors generally  are  more  ready  to  point  to  job  areas  where  they  hope  that 
the  training  will  be  applied  than  to  document  specific  needs  for  the 
training. 


Exhibit  8 

Percentage  Distribution  of  Responses 
on  Questions  Pre-3  and  5 
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A two-tailed  sign  test  was  significant  at 


the  3%  level  of  confidence. 
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Question  Pre-6:  IS  THERE  ANY  OTHER  ADVANCE  INFORMATION  THAT  YOU  THINK 
SHOULD  HAVE  BEEN  PROVIDED  ABOUT  THE  COURSE? 


Level  JL  - Any  answer  that  indicates  no  wish  for  additional  information. 

Leve 1 £ - Any  answer  that  indicates  need  for  further  information. 

Seventy-three  percent  of  supervisors  and  62%  of  participants  con- 
sidered the  advance  information  adequate;  27%  and  38%,  respectively, 
felt  that  it  was  insufficient.  In  this  connection  it  might  be  pointed 
out  that  there  is  more  advance  information  available  concerning  the 
Kepner-Tregoe  course  than  there  is  for  many  training  programs  in  which 
government  agencies  participate.  Evidently,  then,  there  must  be  some 
breakdowns  in  communication  along  the  line.  This  is  best  illustrated 
by  quoting  a few  answers  to  Question  Pre-6 * 

Participants:  "I  would  like  to  have  had  clear  understanding  that  the 

course  will  benefit  staff  people*  Decision-making  can 
be  construed  as  applying  primarily  or  even  exclusively 
to  line  administrators." 

"I  really  received  no  advance  information  that  I can  re- 
call* Should  have  gotten  summary  of  previous  trainees' 
impressions  plus  more  on  actual  content  of  course* 

Chapter  I of  course,  sent  to  me  in  advance,  gave  impres- 
sion course  might  be  too  'business  oriented'  for  my  needs, 
and  I almost  withdrew" 

"Course  outlines  explained  by  training  officers  periodi- 
cally (to  group)  would  help  one  select  training  oppor- 
tunities b^  priority  better*  To  change  mental  gears 
quickly  when  a bulletin  comes  by  is  fraught  with  error, 
disinterest  or  misunderstanding*" 

Supervisors:  "I  believe  it  would  be  helpful  to  give  a more  detailed 

outline  of  what  is  included  in  the  course*" 

"Cannot  say,  since  I have  not  seen  any  'advance  infor- 
mation*' I knew  of  the  course  by  hearsay*" 

"I  have  never  been  provided  anything  in  the  way  of 
advance  information  on  what  the  course  is  other  than  a 
'Problem-Solving  and  Decision-Making  Course.'  To  make 
a full  response  to  this  type  of  questionnaire,  should 
know  more  about  the  course*" 

"What  is  meant  by  'other0  advance  information?  I don't 
know  what  was  provided." 
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"I  would  have  appreciated  information  on  the  objectives  of 
the  course,  its  content,  how  it  was  to  be  conducted,  and 
the  qualifications  of  the  instructors.  Had  I known  all 
this,  I might  not  have  recommended  this  employee  take  it." 
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Question  Pre-7:  WHO  PARTICIPATED  IN  DECIDING  THAT  YOU/THIS  EMPLOYEE 
WERE/WAS  TO  ATTEND? 


Leve 1 2 - Answers  that  explicitly  mention  the  employee  as  participating 
in  the  decision. 

Leve 1 l - Answers  which  indicate  definite  knowledge  of  participation  by 
others  than  employee,  but  do  not  explicitly  mention  the  employee. 

Leve  1 ()  - Any  answer  which  indicates  mere  supposition,  lack  of  knowledge 
on  this  point,  or  no  participation  by  others. 


Exhibit  9 


Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-7 


Exhibit  9 shows  very  close  agreement  between  supervisors  and  parti- 
cipants. Eleven  percent  of  the  participants  were  unable  to  say  who  had 
taken  part  in  the  decision  that  they  should  receive  training.  All  of 
these  individuals,  in  answer  to  Pre-Question  4,  said  that  they  had  not 
discussed  the  course  with  their  supervisors. 

The  fact  that  the  participant  himself  was  not  explicitly  mentioned 
in  answer  to  Question  Pre-7  does  not  necessarily  mean  that  he  did  not 
take  part  in  the  decision  concerning  his  taking  the  course.  In  fact, 
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five  individuals  who  said  that  they  had  participated  in  the  decision  were 
not  mentioned  by  their  supervisors.  Two  more  were  mentioned  by  their 
supervisors  but  not  by  themselves. 
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Question  Pre-8:  OTHER  COMMENTS? 


Leve  1 _1  - If  any  comments  are  given. 

Level  0 - If  no  comments  are  given. 

Thirty-eight  percent  of  the  participants  and  257.  of  the  supervisors 
offered  further  comments.  No  single  theme  stood  out  sharply  in  the  re- 
marks of  either  group.  Participants  expressed  the  hope  that  the  course 
would  prove  profitable;  several  enlarged  upon  benefits  that  they  hoped  to 
gain  from  it.  One  or  two  supervisors  expressed  similar  ideas;  others  com- 
plained that  the  questionnaire  had  not  reached  them  earlier. 

The  results  obtained  from  the  two  pre-training  questionnaires  have 
implications  which  themselves  are  best  stated  as  questions.  Thus,  we  may 
ask: 


1.  Is  there  sufficient  communication  between  training  staff  and 
line  supervisors  concerning  the  content  and  objectives  of  available 
courses?  Is  the  supervisor  usually  provided  with  enough  information  to 
enable  him  to  make  the  best  choices  among  courses  and  among  potential 
trainees? 

2.  If  a supervisor  is  unable  to  document  specific  training  needs  or 
to  specify  ways  in  which  he  expects  his  employees  to  apply  the  proposed 
training,  on  what  basis  does  he  select  trainees?  Is  too  heavy  reliance 
placed  upon  the  "quota"  method  of  assignment?  Can  some  better  methods  of 
selecting  participants  be  found?  Is  the  mere  hope  that  an  employee  will 
be  "helped"  by  a course,  about  which  the  supervisor  knows  little  more  than 
its  title,  sufficient  justification  for  the  cost  of  the  training? 

3.  Is  the  employee  himself  generally  allowed  a voice  in  the  selec- 
tion of  training?  Do  supervisors  sufficiently  discuss  the  proposed  train- 
ing with  their  employees? 
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Post-Training 

The  post-training  questionnaire  for  supervisors  contains  three  ques- 
tions including  "other  comments."  The  participant’s  questionnaire  con- 
tains five,  two  of  which  have  two  parts  each,  making  seven  questions  in 
all..  In  the  following  presentation,  results  for  both  supervisors  and 
participants  will  be  discussed  together  for  three  of  the  questions.  The 
remaining  ones  apply  to  participants  alone. 
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Question  Post-1:  IN  WHAT  WAYS,  IF  ANY,  HAVE  YOUR  IDEAS  CHANGED  AS  TO 

THE  OBJECTIVES  OF  THE  COURSE?  (Supervisors  and 
Participants) 


Leve 1 2 - Responses  enumerating  specific  instance(s)  of  new  or  enlarged 
ideas  of  objectives, 

"My  ideas  have  enlarged  rather  than  changed.  In  addition  to  the 
four  objectives  I gave  before,  I should  now  add:  (1)  to  learn 
how  to  assign  priorities  to  problems  awaiting  solution;  (2)  to 
know  how  to  establish  controls  in  order  that  the  decision  may  be 
carried  out  in  the  proper  manner;  (3)  to  be  able  to  develop  alter- 
natives as  an  adjunct  to  decision-making." 

"Yes.  I originally  thought  of  the  course  in  terms  of  decision- 
making only.  Now  I consider  that  the  objectives  are:  (1)  to 
consciously  and  systematically  analyze  each  problem  situation; 

(2)  to  assess  causes  of  problems  and  systematically  arrive  at 
decisions ." 

Level  - Statement  deals  in  generalities:  Ideas  have  changed,  but 
respondent  does  not  specify  particular  objectives  concerning  which  his 
thinking  has  undergone  change. 

"I  did  not  have  any  specific  ideas  as  to  the  objectives  of  the 
course  before  attending  it,  I now  understand  the  objectives  and 
consider  them  worthy  and  attainable,  at  least  in  part.  The  course 
was  more  concrete  and  applicable,  with  less  pure  theory  than  I 
would  have  expected." 

"I  believe  that  the  rules  and  principles  taught  in  this  course  are 
not  as  specific  as  I had  expected;  that  is,  the  course  may  improve 
abilities  along  these  lines,  but  will  not  replace  ability." 

Level  0_  - Responses  indicating  that  ideas  have  not  changed. 
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Exhibit  10 


Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Post-1 
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The  responses  of  the  two  groups  are  compared  in  Exhibit  10.  Of 
the  supervisors,  21 % reported  changes  in  their  ideas  as  to  objectives 
of  the  course  (Levels  1 and  2 combined),  whereas  58%  of  the  partici- 
pants reported  change.  This  is  to  be  expected  since  the  participants 
were  the  ones  who  had  actually  been  exposed  to  the  training.  If  a 
supervisor  had  not  discussed  the  course  with  his  employee  since  the 
latter's  return,  he  naturally  would  have  had  no  opportunity  to  change 
his  ideas. 

On  the  other  hand,  some  individuals--both  participants  and  super- 
visors— who  reported  "no  change,"  explained  that  they  had  had  a reason- 
ably clear  idea  of  the  objectives  to  begin  with.  A few  supervisors  had 
gained  their  ideas  from  first-hand  experience,  having  previously  taken 
the  course  themselves. 
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Question  Post-2a:  GIVE  EXAMPLES  OF  SITUATIONS  IN  YOUR  OWN  WORK  IN  WHICH 

THE  CONCEPTS  AND  METHODS  TAUGHT  IN  THIS  COURSE  WOULD 
BE  APPLICABLE.  (Participants  only) 


Level  2 - Response  mentions  specific  task(s)  or  problem(s). 

"Carrying  out  instructions  of  superiors  when  (1)  superior  gives  no 
instructions  as  to  method  of  operation;  (2)  more  information  is 
needed  in  order  to  make  plans." 

"Getting  more  out  of  meetings:  How  to  ask  good  questions  to  obtain 
information  needed." 

Level  2.  - Reference  to  general  task  or  problem  area,  but  without  specific 
examples. 

"I  believe  the  concepts  and  principles  taught  in  this  course  could 
be  used  effectively  in  analyzing  policy  change  proposals  and  subse- 
quent directive  issuances." 

"I  feel  it  is  an  excellent  approach  to  identify  problems  relating 
to  the  use  of  operating  forms." 

Level  0 - No  situations  or  problems  mentioned. 

"In  my  work  in  research  the  concepts  and  methods  are  applicable  in 
problem  analysis  phases  and  the  day-to-day  operational  aspects." 

"The  principles  involved  are  associated  in  just  about  every  move  I 
make  in  the  area  of  policy  making  and  decisions  affecting  Federal 
Government  and  public  interest  in  our  Service  programs." 

This  question  is  one  of  several  that  were  introduced  following  a 
revision  of  the  post-training  questionnaire;  consequently,  it  was  not 
asked  of  all  participants.  It  is  essentially  a repetition  of  pre- 
training Question  3.  Changes  in  the  distribution  of  responses  obtained 
before  and  after  training,  therefore,  can  tell  us  something  as  to  the 
effectiveness  of  the  course,  since  the  ability  to  recognize  possible 
applications  is  one  criterion  of  learning  in  this  case. 
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COMPARISON  OF  QUESTIONS  PRE-3  AND  QUESTION  POST- 2a 


Forty-one  participants  answered  both  questions.  Exhibit  11  shows 
the  percentage  distributions  of  their  "before"  and  "after"  responses. 
The  shift  is  impressive  and  highly  significant  statist ically. 5 Thus, 
following  training,  46%  of  the  group  were  able  to  suggest  specific 
applications  in  their  regular  jobs  as  compared  with  only  26%  before 
training.  Similarly,  the  proportion  of  participants  who  were  unable 
to  visualize  any  useful  applications  of  the  course  was  cut  from  347, 
to  14%  following  training. 


Exhibit  11 

Percentage  Distributions  of  Responses  by  Participants 
to  Questions  Pre-3  and  Post  2a 
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^A  two-tailed  sign  test  was  significant  at  the  0.2%  level  of 
confidence. 
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Question  Post-2b:  GIVE  EXAMPLES  OF  SITUATIONS  IN  WHICH  THEY  WOULD  NOT  BE 

APPLICABLE,  (Participants  only) 


Leve  1 1^  - Response  mentions  one  or  more  situations. 

Leve 1 £ - Respondent  can  think  of  no  such  situation,  or  indicates  that 
concepts  and  methods  would  be  applicable  in  all  situations. 

In  their  responses  to  this  question,  42  participants  split  almost 
50-50  Fifty-two  percent  were  able  to  indicate  situations  in  which  they 
felt  the  methods  of  problem  analysis  and  decision-making  taught  in  this 
course  would  not  be  applicable.  The  remaining  48%  were  unable  to  think 
of  any  such  situation-- largely  because,  as  many  respondents  explained, 
they  felt  that  the  methods  would  be  applicable,  at  least  to  some  degree, 
in  any  situation.  It  is  interesting  to  note  in  passing  that,  of  22  indi- 
viduals who  mentioned  specific  situations  or  types  of  problem,  9 singled 
out  personnel  work  as  a field  in  which  the  Kepner-Tregoe  approach  would 
not  apply. 
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Question  Post-3a:  CAN  YOU  RECALL  ANY  SITUATION  DURING  THE  PAST  SIX  MONTHS 

OR  SO  THAT  YOU  MIGHT  HAVE  HANDLED  DIFFERENTLY  IF  YOU 
HAD  ALREADY  TAKEN  THE  COURSE?  (Participants  only) 


Leve 1 2 - Responses  which  mention  specific  situation(s) . 

Level  1_  - Affirmative  answer,  but  no  situation  mentioned.  Note,  however, 
that  situation(s)  may  be  mentioned  in  reply  to  Question  3b,  in  which  case 
a score  of  2 is  assigned  to  3a, 

Level  0 - Negative  answer. 

Fifty-two  participants  replied  to  this  question,  the  percentages  of 
Level  2,  1,  and  0 responses  being  38%,  24%,  and  38%  respectively,  When 
categories  2 and  1 are  combined,  it  appears  that  62%  recalled  situations 
which  they  might  have  handled  differently  if  they  had  taken  the  course 
sooner. 
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COMPARISON  OF  POST-QUESTIONS  2a  AND  3a 


A comparison  of  participants’  responses  to  Questions  2a  and  3a 
yields  some  interesting  findings.  Levels  1 and  2 have  been  combined  to 
produce  Exhibit  12  as  shown  below. 


Exhibit  12 

Distribution  of  Responses  to  Questions  2a  and  3a 
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From  Exhibit  12  it  appears  that,  following  training: 

50%  of  the  participants  were  able  both  to  specify  applications  of 
the  training  and  to  cite  situations  which  they  would  have  handled 
differently; 

36%.  indicated  possible  applications  but  were  unable  to  recall  any 
situations  that  they  would  have  handled  dif ferently--at  least,  not 
during  the  past  six  months; 

7%  were  unable  to  indicate  applications,  but  could  recall  situa- 
tions that  they  would  have  handled  differently  (clearly  an 
inconsistency) ; 

7%  were  unable  to  indicate  either  applications  or  situations  of 
the  type  mentioned. 

Thus,  accepting  the  questionnaire  information  at  its  face  value,  we 
find  unequivocal  evidence  of  payoff  in  half  the  participants.  The  ques- 
tion is  "Is  this  a satisfactory  return  on  the  training  investment?"  A 
more  complete  basis  of  judgment,  of  course,  must  await  the  returns  from 
the  six-month  follow-up.  It  may  well  be  asked,  however,  whether  more 
discriminating  selection  and  preparation  of  participants  might  not  improve 
the  picture,  even  at  this  stage. 
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Question.  Post -3b.  IF  YES,  IN  WHAT  WAYS  DO  YOU  THINK  THE  OUTCOME  MIGHT 

HAVE  BEEN  DIFFERENT?  GIVE  SPECIFIC  EXAMPLES „ 


Leve 1 2 - Specific  example  of  way(s)  in  which  outcome  might  have  been 
changed  for  better, 

"The  three  alternate  methods  of  reporting  performance  and  claiming 
reimbursement  in  the  Special  X Program  might  have  been  reduced  to 
one  standard  method 

"In  the  area  of  procurement,  we  had  a crash  program  in  which  we 
should  have  anticipated  places  where  it  could  go  wrong  and  have 
been  ready  with  back-up  alternatives,  and  have  established  effec- 
tive controls," 

Leve 1 1 - General  statement--specif ic  outcome  not  detailed, 

"Results  may  have  been  approximately  the  same,  but  project  propo- 
sals would  have  been  better  prepared  and  easier  to  follow  up," 

"Now  that  the  course  is  over,  I believe  I would  have  felt  greater 
assurance  of  the  reliability  of  the  decisions,.  The  steps  leading 
up  to  the  final  decision  would  have  been  made  more  ! consciously ' „" 

Leve 1 0 - No  explanation  as  to  how  outcome  would  have  been  different. 

Thirty-two  participants  responded  at  either  Level  1 or  Level  2 on 
Question  Post-3b;  thereby  indicating  some  situation  or  problem  area  that 
they  might  have  handled  differently  after  having  completed  the  Kepner- 
Tregoe  course  Of  these  32  individuals; 

50%  were  able  to  specify  in  some  detail  the  kind  of  improved 
outcome  that  might  have  been  expected; 

38%  were  able  to  suggest  the  probable  outcome  in  more  general 
terms; 

12%  failed  to  hazard  a guess  on  this  point. 
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Question  Post-4:  WHAT  PLANS  DO  YOU  HAVE  FOR  DISCUSSING  THIS  COURSE  WITH 

YOUR  SUPERVISOR?  (Participants) 

IN  WHAT  WAYS  DO  YOU  PLAN  TO  FOLLOW  UP  ON  THIS  EMPLOYEE'S 
USE  OF  WHAT  HE  LEARNED  IN  THE  COURSE?  (Supervisors' 
Question  2) 


Level  2 - Statement  mentions  content  of  discussion  as  planned  or  already 
held,  or  indicates  presentation  to  staff  meeting. 

"Have  touched  on  K-T  program  at  PM  staff  meeting  last  week.  Have 
spent  45  minutes  briefing  X,  Service  Training  Officer,  on  my  reac- 
tions to  program.  I expect  to  visit  in  detail  with  my  supervisor, 
Director  Y,  in  next  week  or  two." 

"He  has  been  urged  to  utilize  all  he  learned  about  decision-making 
in  the  K-T  Course  in  dealing  with  employee  problems  as  well  as 
deciding  permissable  tolerances  for  map  construction  accuracy.  We 
shall  require  that  his  recommendations  for  action  on  employee  pro- 
motions, transfers  and  welfare  as  well  as  for  mapping  tolerances 
show  evidence  that  he  and  his  staff  have  arrived  at  the  decision  by 
objective  and  discerning  analysis.  He  will  be  required  to  analyze 
his  decisions  for  the  Division  Staff." 

Level  _1  - General  statement  with  no  specific  details  or  examples  given. 

"I  plan  to  discuss  the  course  with  him,  perhaps  at  some  length,  as 
office  or  off-duty  time  permit." 

"I  plan  to  work  closely  with  him  in  applying  these  procedures  to 
our  management  problems  whenever  such  procedures  are  suitable  and 
applicable." 

Level  £ - No  discussion  planned. 

"I  do  not  intend  to  develop  a plan  for  follow-up  on  the  employee. 

I do  intend  to  observe  more  closely  the  quality  of  the  employee's 
completed  staff  assignments." 

"Will  not  see  him  before  January  4 and  have  not  discussed  course 
with  him  since  it  was  completed." 


Exhibit  13 


Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Questions  Post-4  and  Post-2 
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As  Exhibit  13  shows,  participants  and  supervisors  showed  good 
general  agreement  in  respect  to  this  question,  and  most  of  them  ind 
cated  some  plans  to  follow-up. 
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Question  Post-5:  OTHER  COMMENTS  (Participants  and  Supervisors) 


Leve 1 1 - If  any  comments  are  given. 

Level.  0 - If  no  comments  are  given. 

Of  participants,  69%,  made  comments;  of  supervisors,  44%,,  It  is 
perhaps  worth  noting  that  both  of  these  percentages  are  higher  than  the 
corresponding  figures  for  the  pre-training  questionnaires  (38%,  and  25% 
respectively),  although,  as  before,  a smaller  proportion  of  supervisors 
than  of  participants  had  comments  to  offer. 

Eleven  supervisors  offered  comments.  Nine  of  these  centered  upon  a 
single  theme--their  subordinates  felt  the  course  to  be  extremely  worth- 
while, Two  respondents  had  acquired  fairly  specific  ideas  as  to  the  values 
of  the  training.  Others  were  more  vague  as,  for  example,  the  mystified 
supervisor  who  said: 

"I  talked  very  briefly  on  the  telephone  with  the  participant  about 
the  course.  He  said  that  it.  was  worthwhile,  but  did  not  specify 
in  what  way  it  was  worthwhile,  I have  now  talked  with  three  men 
who  have  had  the  course  and  they  have  all  said  that  it  was  worth- 
while, but  did  not  tell  me  why,  I'm  intrigued  by  this  'secrecy', M 

The  participants,  as  might  be  expected,  were  much  more  varied  in 
their  comments  Among  the  points  most  frequently  mentioned  were  the 
fol lowing: 

24%  praised  their  instructor; 

21%  felt  that  the  course  should  be  given  at  an  earlier  stage 
in  the  individual’s  career  development; 

14%  felt  that  more  time  should  be  allotted  to  the  training; 

10%  agreed  that  the  decision-making  part  of  the  course  was 
less  effective  than  the  part  dealing  with  problem  analysis. 
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V.  Discussion 


Results  presented  in  the  last  chapter  provide  an  incomplete  evalu- 
ation of  the  Kepner-Tregoe  course.  The  real  test  will  come  with  the 
f ollow-up--this  will  give  some  indication  of  the  extent  to  which  parti- 
cipants have  put  the  concepts  and  methods  to  use  in  their  jobs.  At  the 
present  stage,  this  much  can  be  said:  If  a Level  1 or  Level  2 answer 
to  both  Post-Questions  2a  and  3a  be  accepted  as  a criterion  of  success, 
then  exactly  half  of  the  42  participants  for  whom  complete  data  were 
available  successfully  completed  the  course.  Six  of  the  42  failed  to 
meet  this  criterion,  and  15  fell  in  the  doubtful  category  (Exhibit  12). 

Is  this  a good  showing? 

On  the  other  hand,  our  findings  up  to  this  point  raise  some  inter- 
esting questions  regarding  the  specific  requirements  of  a sound  training 
program  as  described  in  Section  II  of  this  report. 

Consider  first  a question  of  communication.  Twenty-seven  percent 
of  the  supervisors  and  38%  of  the  participants  felt  that  the  advance 
information  provided  them  concerning  the  course  was  insufficient.  How, 
then,  could  they  arrive  at  a sound  decision  concerning  the  objectives  of 
the  course  or  its  value?  Fortunately,  in  this  case  the  course  happened 
to  be  a good  one;  but  will  this  always  be  true?  Would  it  not  strengthen 
the  overall  training  effort  if  course  announcements  were  screened  for 
adequacy  of  information  before  they  were  passed  along  to  agency  training 
officers  and  supervisors?  A brief  memorandum  calling  attention  to  this 
point  might  well  be  appended  to  the  announcement;  on  the  basis  of  the 
adequacy  of  information  the  course  might  be  recommended  or  not  recom- 
mended for  consideration.  Would  this  be  desirable? 

Another  facet  of  the  communication  problem  appears  in  the  fact  that 
in  approximately  one-half  of  the  cases  no  discussion  between  supervisor 
and  employee  occurred  prior  to  the  latter's  participation  in  the  Kepner- 
Tregoe  course.  Of  course-~as  our  own  data  show--prior  discussion  with 
one's  supervisor  is  not  an  essential  condition  of  learning.  Nevertheless, 
some  communication  between  employee  and  supervisor  would  seem  to  be  desir- 
able from  the  standpoint  of  attention  to  training  needs,  a wise  selection 
of  training  programs,  and  the  integration  of  training  with  operations. 

Is  it  indicative  of  cooperative  planning  by  supervisor  and  employee  when 
a participant  writes,  "I  have  no  idea  what  my  supervisor's  ideas  are"? 

It  was  suggested  in  the  second  chapter  that  the  systematic  use  of 
an  evaluation  procedure  such  as  this,  should  lead  eventually  to  better 
results  from  the  training  effort.  The  evaluation  technique  described  in 
this  report  is  now  being  extended  to  two  other  training  programs:  Seminars 
in  Executive  Development  (SED)  and  the  newly  inaugurated  Seminar  in  Middle 
Management  (SIMM),  both  of  which  are  interagency  programs.  In  the  case  of 
SIMM,  the  proposed  evaluation  is  mentioned  in  the  brochure  announcing  the 
program. 
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Although  it  is  desirable,  it  is  not  always  necessary  to  make  as 
detailed  an  analysis  of  results  as  was  done  in  this  case.  The  full-scale 
treatment  of  evaluation  data  was  offered  here  to  show  how  it  can  be  done. 
But  careful  examination  of  the  questionnaire  responses  alone,  without 
formal  content  analysis  or  coding,  can  yield  significant  information 
about  the  effectiveness  of  the  overall  training  effort.  The  main  aim  is 
to  provide  for  specific  plans  oriented  to  individual  development  needs 
measured  against  defined  management  objectives,  to  provide  for  shared 
understanding  of  training  purposes,  to  focus  on  specific  behavioral  ef- 
fects of  training  and  upon  the  use  of  such  performance  evidence  in  evalu- 
ating training--in  short,  to  create  situational  conditions  facilitating 
the  effective  utilization  of  training  received. 

To  realize  the  potential  benefits  of  evaluation,  however,  it  is  nec- 
essary that  the  procedure  be  carried  out  methodically  and  on  a continuing 
basis.  This  means,  among  other  things,  insisting  upon  complete  returns 
from  both  participants  and  their  supervisors  on  all  questionnaires.  Un- 
less this  requirement  is  observed,  the  entire  effort  becomes  pointless. 

If  management  itself  treats  the  evaluation  casually,  others  will  do  so 
too.  The  result  can  be  an  altogether  unwarranted  complacency  regarding 
the  effectiveness  of  the  entire  training  and  management  development 
program. 
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TRAINING  RESEARCH  QUESTIONNAIRE 

INSTRUCTIONS  FOR  PARTICIPANTS 

A considerable  number  of  people  in  the  Department  have  taken  or 
soon  will  participate  in  the  Kepner-Tregoe  training  program  in 
Management  Problem  Analysis  and  Decision-Making.  In  order  to  find  out 
what  values  this  training  may  have  for  our  employees,  it  is  necessary 
to  obtain  some  information  before  as  well  as  after  you  have  attended 
the  course. 

In  answering  the  accompanying  questions,  please  be  as  specific  as 
possible.  When  you  have  completed  the  course,  you  may  be  asked  some 
more  questions  about  it.  We  also  plan  to  follow  up  from  time  to  time 
in  the  future  to  check  on  long-range  benefits  from  the  program. 

The  most  valuable  information  is  your  frank  reactions  based  upon 
your  personal  expectations,  needs,  experience  and  applications.  So, 
work  independent ly- -do  not  consult  anyone  else  in  answering  this 
questionnaire.  Feel  free  to  react  favorably  or  unfavorably. 

Two  blank  sheets  are  provided  for  your  responses  (feel  free  to 
use  additional  sheets  if  you  need  them) „ Please  number  your  answers 
to  correspond  with  the  questions. 


Thank  you. 
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Questions  for  Participants  Prior  to  Training 

1.  What  do  you  think  are  the  specific  objectives  of  the  course? 

2.  How  does  this  seem  to  fit  into  your  own  career  plans? 

3.  On  the  basis  of  information  available  to  you  now,  in  what  particular 
ways  do  you  expect  to  apply  this  training  in  your  future  work?  Give 
examples . 

4.  Have  you  discussed  points  1,  2,  and  3 with  your  supervisor?  If  Yes, 
do  you  feel  that  he  shares  the  same  expectations  as  you  do?  If  not, 
how  do  your  expectations  and  his  differ  as  you  see  it? 

5.  Looking  back  over  your  work  during  the  past  six  months  or  so,  has 
anything  happened  to  bring  home  to  you  a need  for  this  kind  of 
training?  Give  actual  examples, 

6.  Is  there  any  other  advance  information  you  think  should  have  been 
provided  about  the  course? 

7.  Who  participated  in  deciding  that  you  were  to  attend? 


8. 


Other  comments. 
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TRAINING  RESEARCH  QUESTIONNAIRE 

INSTRUCTIONS  FOR  SUPERVISORS 

A considerable  number  of  people  in  the  Department  have  taken  or  soon 
will  participate  in  the  Kepner-Tregoe  training  program  in  Management 
Problem  Analysis  and  Decision-Making,  In  order  to  find  out  what  values 
this  training  may  have  for  our  employees,  it  is  necessary  to  obtain  some 
information  before  as  well  as  after  they  have  attended  the  course. 

Please  answer  the  accompanying  questions  for  each  of  your  employees 
who  is  about  to  take  the  course.  In  giving  your  answers,  please  be  as 
specific  as  possible.  When  your  employee(s)  have  completed  the  course, 
you  may  be  asked  some  more  questions.  We  also  plan  to  follow  up  from 
time  to  time  in  the  future  to  check  on  any  long-range  benefits  from  the 
program. 

The  most  valuable  information  is  your  frank  reactions  based  upon 
your  personal  experience  with  your  employees,  including  your  needs  and 
expectations  concerning  them.  So  work  independent ly--do  not  consult 
anyone  else  in  answering  this  questionnaire.  Feel  free  to  react 
favorably  or  unfavorably. 

Two  blank  sheets  are  provided  for  your  responses  (feel  free  to  use 
additional  sheets  if  you  need  them).  Please  number  your  answers  to 
correspond  with  the  questions. 

Thank  you. 
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Questions  for  Participants 1 Supervisors  Prior  to  Training 

1.  What  do  you  think  are  the  specific  objectives  of  the  course? 

2.  How  does  this  fit  into  your  Agency  Training  and  Development  Program 
for  this  participant? 

3.  On  the  basis  of  information  available  to  you  now,  in  what  particular 
ways  do  you  expect  this  employee  to  apply  the  training  in  his  future 
work?  Give  examples. 

4.  Have  you  discussed  points  1,  2,  and  3 with  the  employee?  If  Yes, 
do  you  feel  that  he  shares  the  same  expectations  as  you  do?  If 
not,  how  do  your  expectations  and  his  differ  as  you  see  it? 

5.  Looking  back  over  his  work  during  the  past  six  months  or  so,  has 
anything  happened  to  bring  home  to  you  his  need  for  this  kind  of 
training?  Give  actual  examples. 

6.  Is  there  any  other  advance  information  you  think  should  have  been 
provided  about  the  course? 

7.  Who  participated  in  decid  ng  that  this  employee  was  to  attend? 


8. 


Other  comments. 
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TRAINING  RESEARCH  QUESTIONNAIRE 

INSTRUCTIONS  FOR  PARTICIPANTS 

Now  that  you  have  completed  the  Kepner-Tregoe  course  in  "Management 
Problem  Analysis  and  Decision-Making,"  we  would  like  to  have  your  frank 
reactions  on  certain  points.  As  explained  in  the  pre-training  question- 
naire which  you  filled  out  earlier,  we  also  plan  to  follow  up  in  the 
future  in  order  to  check  on  long-term  effects  of  the  program. 

Feel  free  to  react  favorably  or  unfavorably  or  to  agree  or  disagree 
with  anyone  else's  point  of  view.  What  we  want  are  your  reactions, 
based  upon  your  own  personal  experience.  In  answering  these  questions, 
please  be  as  specific  as  possible. 

As  before,  two  blank  sheets  are  provided  for  your  responses  (do  not 
hesitate  to  use  additional  sheets  if  you  need  them)  „ Please  number  your 
answers  to  correspond  with  the  questions. 


Thank  you  again. 
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Questions  for  Participants  after  Training 

1.  In  what  ways,  if  any,  have  your  ideas  changed  as  to  the  objectives 
of  the  course? 

2.  a.  Give  examples  of  situations  in  your  own  work  in  which  the  con- 

cepts and  methods  taught  in  this  course  would  be  applicable, 
b.  Give  examples  of  situations  in  which  they  would  not  be  applicable. 

3.  a.  Can  you  recall  any  situations  during  the  past  six  months  or  so 

that  you  might  have  handled  differently  if  you  had  already  taken 
the  course? 

b,  If  yes , in  what  ways  do  you  think  that  the  outcome  might  have 
been  different?  Give  specific  examples. 

4.  What  plans  do  you  have  for  discussing  this  course  with  your  supervisor? 


5.  Other  comments. 
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TRAINING  RESEARCH  QUESTIONNAIRE 

INSTRUCTIONS  FOR  SUPERVISORS 

Now  that  your  employee  has  completed  the  Kepner-Tregoe  course  in 
"Management  Problem  Analysis  and  Decision-Making,"  we  would  like  you  to 
answer  two  more  questions.  As  explained  in  the  pre-training  questionnaire 
which  you  filled  out,  we  also  plan  to  follow  up  in  the  future  in  order  to 
check  on  long-term  effects  of  the  program. 

Please  answer  the  questions  below  for  each  employee  who  has  just 
finished  the  course.  You  probably  will  want  to  talk  to  the  employee 
before  you  start.  However,  feel  free  to  react  favorably  or  unfavorably 
or  to  agree  or  disagree  with  anyone  else's  point  of  view.  What  we  want 
are  your  reactions,  based  upon  your  own  experience  and  observations.  In 
giving  your  answers,  please  be  as  specific  as  possible. 

As  before,  two  blank  sheets  are  provided  for  your  responses  (do  not 
hesitate  to  use  additional  sheets  if  you  need  them).  Please  number  your 
answers  to  correspond  with  the  questions. 

Thank  you  again. 


Questions 

1.  In  what  ways,  if  any,  have  your  ideas  changed  as  to  the  objectives 
of  the  course? 

2.  In  what  ways  do  you  plan  to  follow  up  on  this  employee's  use  of 
what  he  learned  in  the  course? 


3.  Other  comments. 


